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ABSTRACT 

In March 1989, the Northwest Regional Educational 
Laboratory entered into a contract with the Nevada Department of 
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state legislation, requires each school district to administer 
examinations in all public schools to determine student achievement 
in reading, writing, and mathematics in grades 3, 6, 9, and bt ^re 
completion of grade 12 (usually grade 11), No formal evaluation of 
the program had previously been conducted. To maximize the chances of 
providing useful information for decision makers, the evaluation 
proposed to- review the literature; conduct survey research to 
collect data from stakeholder groups; maintain an orientation toward 
program improvement; and use cost efficient methods. Three general 
areas were identified for the evaluation: (1) similarities and 
differences between the Nevada program and comparable activities in 
other states; (2) program administration issues; and (3) program 
improvement issues. Descriptive statistical methods were planned for 
analysis of the data collected. The evaluation was scheduled for the 
fall of 1989, prior to a planned changeover in the tests used to 
determine proficiency. Appendices comprising half the document 
contain survey forms and interview questions planned for the 
evaluation. (SLD) 
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I. INTRODUCTION 



In March 1989, the Northwest Regionai Educational Laboratory (NWREL) entered into a contractual 
arrangement with the Nevada Department of Education (NDE) to develop an evaluation design for the Nevada 
Proficiency Examination (NPE) Program. The design is to enable NDE staff to implement a comprehensive 
evaluation of the program in the fall of 1989. NWREL may also provide technical assistance in implementing 
the design and in using evaluation results to improve the NPE program. 

Following the contract award, NWREL staff met with the NPE staff in March to discuss general issues relating 
to the proposed evaluation. A followup conference call in April further clarified primary purposes of the 
evaluation and related concerns. NWREL staff subsequently developed a preliminary draft of an evaluation 
design and draft versions of instruments to be used in the proposed evaluation. 

Concjrrently, the NPE staff assembled a panel of individuals representing the various relevant "stake-holder" 
and policy-making groups. The panel was charged with the task of reviewing the draft materials and providing 
feedback to NWREL staff for revising and refining the evaluation design and related instruments. The 
following individuals served on the Nevada Proficiency Examination Program Evaluation Advisory Panel: 

Patricia Boyd 

Nevada Department of Education 
Jeanne Bolts 

Nevada Legislative Council Bureau 
Janice Clark 

Nevada State Board of Education 
Kevin Crowe 

Nevada Department of Education 
Fred Doctor 

Washoe County School District 
Mark Lange 

Clark County School District 
Karen Ostrow 

Douglas County School Board 
Eugene Paslov 

Nevada Department of Education 
Mavis Scarff 

Nevada Department of Education 
Robert Scott 

Carson City School District 
Donald Williams 

Nevada Legislative Council Bureau 

The advisory panel met with NWREL staff at the Nevada Department of Education on May 23, 1989 to provide 
input on the evaluation design and related instruments. This document presents the revised design and 
instruments. Dui ing the implementation of the evaluation study, the advisory panel will be reconvened to 
provide further input, as needed. 
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II. THE NPE PROGRAM 



The Nevada Proficiency Examination (NPE) Program was established in 1977 by legislative mandate in 
response to accountability concerns. The law, as amended by the 1987 legislature, requires the board of 
trustees of each school district to administer examinations in all public schools to determine student 
achievement and proficiency in reading, writing and mathematics. The examinations are to be given before the 
completion of grades 3, 6, 9 and 12. The law further stipulates that students failing to demonstrate adequate 
achievement in grades 3, 6 or 9 must receive remedial Instruction to help them achieve proGciency. Students 
failing to pass the high school proficiency examination may be given a certificate of attendance, in place of a 
diploma, if they have reached the age of 17 years. The law also requires the submission of the results of 
proficiency examinations by each board of trustees to the state superintendent of public instruction. Such 
submissions are to be made in accordance with regulations prescribed by the state board of education. 

In compliance with the legislative mandate, competency statements were developed in reading, writing and 
mathematics by work groups consisting of teachers, school administrators, business leaders and legislators. 
These competency statements were subsequently adopted by the State Board of Education and, with minor 
modifications over the years, served as the basis for developing the various proficiency tests at the high school 
level. 

In 1984, the State Board of Education adopted a reused set of competency statements in the area of writing. It 
also adopted the use of a scale score of 370 as a standard for passing the high school proficiency tests in reading 
and mathematics. This new standard provided a passing rate of 90% for regular students tested in the spring of 
1985. 

The current program focuses on testing in grades 3, 6, 9 and 11. The Comprehensive Tests of Basic Skills 
(CTBS) is administered to students in grades 3, 6 and 9. High school proficiency tests developed by Nevada are 
administered to students in grade 11. Students must demonstrate proficiency in reading, writing and 
mathematics to receive a high school diploma. 

Proficiency test data are aggregated up to the state level. Data summaries are also prepared for each school 
district or county. Norm-referenced test data consist of percentages of students in low, middle and high stanine 
groupings for each grade and subject area. For the high school proficiency tests, passing rates and average 
scores are reported. In addition, the data include percentages of high school students getting 50 percent or 
more of the items correct in each subtest or skill area. 

Regulations have recently been drafted to further clarify what tests are required, test administration dates, and 
reporting procedures. 

The program is supported by funds which the districts receive as part of their budgets. The State Department 
of Education, which manages the program, then recovers testing costs from the districts. 

Although the NPE Program has been in existence for over a decade, no fampl evaluation has been cor ducted 
for the program. 
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III. DESIGN CONSIDERATIONS 



The evaluation of an educational intervention as complex as a statewide testing program will of necessity be 
primarily descriptive and formative. Since the NPE Program is an existing and continuing activity, it is not 
possible to use any true or quasi-experimental designs in the evaluation. Instead, we will focus in on other 
considerations to maximize chances of providing useful information for decision-makers. 

Litetature Review . A thorough review of the extant literature on competency testing and statewide assessment 
activities (e.g., NWREL, 198J) will provide information on the similarities and differences between the NPE 
Program and comparable activities implemented in other states. In addition, much can be learned with respect 
to advantages and disadvantages of various approaches to implementing accountability systems on a statewide 
basis. This activity will include a review of records (e.g., data Pies) maintained at the Nevada Department of 
Education and/or the various school distnet offices. 

Survey Research. We will depend heavily on survey research methodology in collecting perceptual data from the 
various "stake-holder" groups. This will include questionnaire surveys of county and school level staff, 
interviews with state level administrators and members of the State Board of Education and the state 
legislature. Interviews or surveys may also be conducted with students and parents on a sampling basis. 

Improvement Orientation. The evaluation will be primarily formative with a strong emphasis on program 
improvement. A primary consideration is to obtain information which will be useful in identifying program 
strengths and weaknesses and in initiating improvement activities. 

£Q$t_Effiqsncy. Throughout the evaluation we will use only the most cost efficient methods of data collection 
and analysis. For example, instruments will be designed for easy administration and tabulatijn of results. 
Onsite interviews will be conducted only if no alternatives exist for collecting comparable information. In such 
events, we will use the most efficient sampling procedures to select the target groups. 
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IV. EVALUATION QUESTIONS 

The current interest in evaluating the NPE Program appears to center around three general areas: 

o Similarities and differences between the Nevada program and comparable activities in other 
states 

o Program administration issues 

o Program improvement issues 

Each of these areas subsumes a number of questions which might be addressed in the proposed evaluation. 
These are listed as follows: 

Program Similarities and Differences 

1. In comparison with other state testing programs, how different is the NPE Program with respect to grade 
level and subject area coverage? 

2. How much emphasis does the program place on norm- referenced measurement versus criterion- 
referenced (mastery of specific skills) measurement? How is the program different from its counterparts 
in other states in this regard? 

3. How different is the program from its counterparts in other states with respect to program governance and 
policy setting? 

4. How different is the program from its counterparts in other stales with respect to (a) program goals, (b) 
staffing, (c) resource allocation, and (d) use of test results? 

5. How different is the program from its counterparts in other states with respect to program impact on high 
school graduation? For example, proportionally how many students are denied a high school diploma 
because they fail the proficiency tests? 



Program A dministration 

1. Should the program be administered entirely by the state? For example, should the budgeting process be 
streamlined so that the state keeps the resources at the state level instead of allocating them to the 
districts and then recovering them from the districts? 

2. Who should be involved in setting policies for the program? What interests should be represented in the 
policy-making process? Wha* role should the districts play in that process? 

3. Who should be responsible for quality assurance? What mechanisms should be put in place to ensure that 
the NPE data are valid and reliable? 

Program Improvement 

1. What are the strong and weak areas in the NPE Program as perceived by the various "stake-holder" 
groups? What can be done to strengthen the weak areas? 

2. Should the program be expanded to other areas (e.g., more subject areas and/or grade levels) to provide a 
more comprehensive picture of student achievement in Nevada? 

3 How should data be used to maximize their impact on curriculum and instruction at the school levels? 

4 
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V. DATA COLLECTION 



Data collection methods will include literature review, questionnaire survey, and onsite interview. 

Literature Review 

This will include a review of the relevant research literature to identify similarities and differences between the 
Nevada program and proficiency testing efforts in other states. Of particular interest will be differences with 
respect to program goals, governance and impact on students. In addition, a document review will be 
conducted of relevant data files maintained at the state and district offices. A primary purpose of the document 
review will be to gather data relating to program impact on high school graduation rates. 

Questionnaire Survey 

frssppndgnt grpypft. Questionnaire surveys will provide the bulk of the evaluation data. The survey respondent 
groups will include the following: 

o Teachers 

o School administrators 

o District administrators 

o District superintendents 

o Members of local school boards 

o Faculty members of post-secondary educational institutions 

o Members of the state legislature 

Sampling . All district superintendents, school administrators, members of local school boards, and members of 
the state legislature will be included in the survey. A stratified random sampling procedure, using county and 
school as stratification factors, will be used to identify district administrators and teachers to be included in the 
survey. Specifically, a 20 percent random sample of teachers will be selected from each school to respond to the 
survey. A 20 percent random sample of district leve 1 administrators will be selected from each of the 17 
districts. In cases where a school or district has fev/er than five teachers or administrators, all teachers or 
administrators will be included in the survey. A random sample of individual teachers or administrators can be 
identified from a staff list by using a table of random numbers or by a random draw from a pool of names. A 10 
percent random sample of faculty members at each post-secondary education institution will be selected to 
respond to the survey. A random sample can be identified from a staff list by using a table of random numbers. 
In cases where a post-secondary education institution employs fewer than 10 teaching staff, all members of the 
teaching staff will be included in the survey. 

Survgy administration. A mail-out procedure will be used to administer the survey to the respondent groups. A 
cover letter should a<xompany the questionnaire to explain the purpose of the survey a^d to provide instructions 
for completing and returning the survey. A self-addressed, stamped envelope should be included for returning 
the completed questionnaire. Also, followup procedures (e.g., letters, phone calls, postcards) should be 
implemented to improve the response rate. 

A revised survey questionnaire and a sample cover letter are included in the appendices. 

Onsite/Telephone Interview 

Respondent grpyp$. This data collection activity will provide a rich and comprehensive data base for program 
improvement. Onsite or telephone interviews will be conducted with the following groups: 

o State level administrators, including the Superintendent of Public Instruction 

o Key members of the State Board of Education 

o Key members of state legislature (e.g., education committee members) 
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o Key members of school-community organizations 
o Business leaders 



Sampling. For each of the respondent groups, a list of names will be identified jointly by the Planning, Research 
and Evaluation Branch and an evaluation advisory group (e.g., the evaluation design review panel). It is 
important that the selected individuals constitute a good representation of the various "stake-holder" groups. 
For example, business leaders should include presidents of chambers of commerce, personnel directors of 
major employers in each county and leaders of other business organizations. Key members of school- 
community organizations should include chairpersons of parent-teacher associations and parent advisory 
councils. With respect to logistics, it is important that the selected individuals be noti r -d of the pending 
interviews in advance. 

CondlKt Qf interviews. We have developed a list of sample questions to be used as a general guide for onsite or 
telephone interviews. Interviewers may reorder the sequence of questions and rephrase or paraphrase the 
questions, whenever appropriate. They may even skip questions that are obviously inappropriate in a particular 
situation. .*lso, interviewers are urged to use probes (e.g., "Why do you think so?") whenever needed to get a 
fuller answer or to let the interviewee expand on a response. 

We strongly suggest that all interviewers attend an orientation and training session before they conduct the 
interviews. Samples of interview questions are included in the appendices. 

Survey of Students and Parents 

To accommodate resource constraints, surveys of students and parents will be conducted in conjunction with 
existing data gathering activities. Specifically, student perceptions will be gathered through a brief survey 
attached to or imbedded in the state developed proficiency tests. Students taking the proficiency tests will be 
asked to complete the survey immediately following the testing session. Parent perceptions will be gathered 
through the bi-annual telephone poll (Needs Assessment: The Opinions of Nevada Voters About Their FubVc 
Schools) scheduled to be conducted by the Nevada Department of Education in March 1990. Survey items 
pertaining to the Proficiency Examination Program will be incorporated in the survey instrument. Taking 
advantage of the existing data collection activities will result in considerable cost savings and at the same time 
provide representative lesults on student and parent attitudes toward the program. 

Table 1 presents the anticipated number of survey responses for each respondent group. Sample surveys for 
students and parents are included in the appendices. 



Table 1 

Anticipated Numbers of Survey Responses by Respondent Group 



Respondent Group 



Number of Responses 



Survey of total population 

Local superintendents 

Local school board members 

Local principals/assistant principals 

Legislators 

Students* 

TOTAL 

Survey of 20% stratified random sample 

Public school teachers 
Local district assistant 

superintendents and 

supervisory personnel 
University of Nevada faculty 



TOTAL 



C. Telephone/onsite interviews 



State level administrators, 

including the Superintendent 

of Public Instruction 
Members of State Board of Education 
Legislators serving on the Education 

Committee 
Parents** 

Presidents of local chambers of commerce 
Other business and community leaders 
to be identified 



TOTAL 

GRAND TOTAL 



17 
104 
427 

54 
11,000 

11,602 



1,739 



30 
243 

2,012 



9 
9 

9 
98 
17 

35 

577 

14,191 



* To be collected during the administration of Nevada Proficiency Examination to all 11th graders in 
spring, 1990. 

** To be collected as part of the Department's bi-annual telephone poll (Needs Assessment: Vie 
Opinions of Nevada Voters About Their Public Schools) in spring, 1990. 
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VL DATA ANALYSIS AND REPORTING 



Most of the evaluation data will be descriptive, A significant portion will be erbal information. Accordingly, 
in most cases, descriptive statistical methods will be used in data analysis. Much of these data will be analyzed 
in terms of frequency counts, percentages and cross-tabulations. However, some of the perceptual data (e.g., 
Likert scale data) will be amenable to inferential statistical analysis. Tests of statistical significance, parametric 
and non-parametric, will be made when warranted. 

In interpreting the data, policy implications will be of greater interest than statistical significance. The primary 
intent will be to enhance our understanding of the NPE Program as a policy tool for strengthening Nevada's 
accountability system. Much of the information can be interpreted in terms of the symbolic and instrumental 
value of such a policy tool (Airasian, 1988; EUwein, ct al., 1988). An overriding objective is to provide 
information that state level administrators will find useful in planning and improving the accountability system. 

Major findings will be presented in a written report which will include an executive summary and a section on 
conclusions and recommendations. The primary purpose of the report is to present information which state 
level decision- makers can use to make program modification and improvement. 
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VII. IMPLEMENTATION 



General Considerations 

The implementation of the evaluation design will require close consultation and cooperation among the NPE 
staff, the advisory group and the evaluator. A close working relationship will ensure the attainment of two 
primary objectives: (a) the evaluation information will be meaningful and useful to state level decision-makers; 
and (b) the evaluation will be of high technical quality so that its results ore valid and reliable. Although the 
design has provided guidelines for instrument development, data collection and analysis, it is likely that some of 
the procedures may need to be adjusted to accommodate reality constraints as the evaluation progresses. Such 
adjustments should be made in close consultation among the advisory group, the NPE staff and the evaluator. 

Timelines 

The evaluation is scheduled for implementation in the fall of 1989. The State Department of Education has 
adopted the Comp thensive Tests of Basic Skills, 4th edition (CTBS/4) as part of the proficiency testing 
activities at grades 3, 6 and 9. It is important that the bulk of the evaluation data be collected prior to the use of 
the new "est to avoid any confounding effects which the change-over might introduce. 

Task List 

A listing of specific tasks and dates necessary for completing the evaluation follows: 

Person Responsible Date 

1. Develop evaluation 
design and survey 
instruments 

2. Review design and 
instruments 

3. Revise design and 
instruments 

4. Review relevant 
research literature 

5. Review existing program 
documents/data files 

6. Develop survey 
mailing list 

7. Provide training 
session for 
interviewers 

8. Administer survey 
questionnaire 

9. Conduct interviews 

10. Conduct followups 
on surveys 

11. Perform data quality 
checks and data 
entry 

12. Begin data analysis 
and interpretation 

13. Collect student 
and parent survey 
data 



NWREL 

Advisory Panel 

NWREL 

NDE 

NDE 

NDE 

NDE 
NDE 

Interviewers 
NDE 

NDE 
NDE 

NDE 



5/89 
5/89 
6/:>9 
9/89 
9/89 
9/89 

10/89 

11/89 
11/89 

12/89 

1/90 
2/90 

4/90 



Task 



Person Responsible 



Date 



14. Complete data 
analysis and 
interpretation 

15. Prepare evaluation 
report 

16. Disseminate evaluation 
results 

17. Initiate program 
improvement activities 



NDE 
NDE 
NDE 
NDE 



5/90 
6/90 
7/90 
Fall/90 



During the implementation of the evaluation study, the Nevada Proficiency Examination Program Evaluation 
Advisory Panel will be convened to provide input and guidance, as appropriate. 
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Sample Cover Letter for Questionnaire Survey 



Dear Colleague: 

In accordance with a legislative mandate, the Nevada Department of Education established in 1977 the Nevada 
Proficiency Examination Program. The program requires each school district to administer examinations in its 
public schools to determine student achievement and proficiency in reading, writing and mathematics at grades 
3, 6- 9 and 11. 

Students who fail to demonstrate adequate achievement are provided with remedial instruction. High school 
seniors are not allowed to graduate without passing the proficiency examinations. If they have reached the age 
of 17 years, they may receive a certificate of attendance in place of a diploma. 

The Nevada Proficiency Examination Program uses a standardized achievement test to assess student 
achievement at grades 3, 6 and 9. Students in grade 11 take state developed proficiency tests in reading, 
mathematics and writing. The program requires each school district to submit results of the proficiency 
examinations to the Superintendent of Public Instruction. 

The Nevada Department of Education is now conducting an evaluation of the program. The evaluation is 
primarily formative, seeking to obtain information which will be useful in identifying program strengths and 
weaknesses and in initiating improvement activities. An important part of the evaluation is a questionnaire 
survey of a v*de range of school-community groups. You have been identified as a membei of one of the 
respondent groups. 

The enclosed questionnaire is being sent to you as a means of obtaining your perceptions regarding the Nevada 
Proficiency Examination Program. The survey consists of structured and open-ended items. It is important 
that you respond to.al! items. Please use the T option if you arc not sure about an item or are unable to 
provide a response. 

We would appreciate your completing the questionnaire and mailing it back to us in the enclosed self- 
addressed, sta nped envelope. Since you are part of a carefully selected random sample, it is extremely 
important that we receive your completed questionnaire. 

Please be assured that your input will help us achieve a comprehensive evaluation of the program. Also, please 
be candid in responding to the survey items. The survey data will be summarized in such a way that you will not 
be identified with your responses. 

If you have questions regarding the survey, please call Dr. Kevin Crowe at (702) 885-3130. We would 
appreciate receiving your completed survey by . 

Thanks very much for your help with this important evaluation pr jject. 
Sincerely, 
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Nevada Proficiency Examination Program 
Survey Questionnaire 

County 

School/Inst i i uwon 

Your Position (check all that apply): 
Teacher 

School administrator 

District administrator 

Local school board member 

University /college faculty member 

State legislator 

Other (spfvify) 

Date 

If you are a school teacher, please indicate grade levels and subject areas which you are currently teaching: 

Grade Levels 

Subject Areas 

The purpose of this survey is to find out how you feel about the Nevada Proficiency Examination Program. We 
want your opinions on how it is being implemented and its impact on schools and students. The survey consists 
of structured items as weU as open-ended ones. Please respond to all items. 

1. There are several groups of people who may have influence on policies governing the Nevada Proficiency 
Examination Program. Some of these are listed in the left-hand column. In the column labeled "NOW," 
circle the numbei which best shows how much influence you think each group actually has now . In the 
column labeled "SHOULD HAVE," circle the number which best shows how much influence you think 
each group ideally should have . Circle H ?" if you are not sure or are unable to give a response. 



Influence of following groups on program policies 

NOW SHOULD HAVE 



Legislature 

State Board of 
Education 

State Department 
of Education 

Testing 
specialists 

District 

curriculum 

specialists 



A 
Great 

None Deal 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 



A 
Great 

None Deal 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 



1 1< 



Influence of following groups on program policies 

NOW SHOULD HAVE 



District 
administration 

Teachers 

Citizen 
committees 

Parents 

Community ac 
large 

Other (specify below) 



A 
Great 

None Deal 

2 3 4 5 ? 

2 3 4 5 ? 

2 3 4 5 ? 

2 3 4 5 ? 

2 3 4 5 ? 

2 3 4 5 ? 



A 
Great 

None Deal 
2 3 4 J 
2 3 4 5 

2 3 4 5 
2 3 4 5 

2 3 4 5 

2 3 4 5 



The Nevada Proficiency Examination Program can serve several purposes. Some of these are listed in the 
left column. In your opinion, how well does the program serve these purposes now? Circle the number 
which best reflects your opinion under the "NOW SERVE" column. Ideally, how wellihojild the program 
serve these purposes? Circle the number which best reflects your opinion under the "SHOULD SERVE" 
column. Circle "?" if you are not sure or are unable to give a response. 



How well program serves following purposes 

NOW SERVE SHOULD SERVE 



Provide remedial/ 

diagnostic 

information 

Provide standards 
for high 

school graduation 

Monitor district 

educational 

programs 

Provide standards 
for grade 
promotion at 
the elementary 
grades 

Provide a basis 
for deciding 
which districts 
should receive 
additional support 
or technical 
assistance 

Provide a basis 
for sanctioning 
low-achieving 
districts 

Collect normative 
data for 
accountability 

Provide information 
to state 
legislature 



Not At 
All 
Well 



Very 
Well 



1 2 3 4 5 ? 



1 2 3 4 5 ? 



1 2 3 4 5 ? 



1 2 3 4 5 ? 



1 2 3 4 5 ? 



1 2 3 4 5 ? 



1 2 3 4 5 ? 



1 2 3 4 5 ? 



Not At 
All 
Well 



Very 
Well 



1 2 3 4 5 ? 



1 2 3 4 5 ? 



1 2 3 4 5 ? 



1 2 3 4 5 ? 



1 2 3 4 5 ? 



1 2 3 4 5 ? 



1 2 3 4 5 ? 



1 2 3 4 5 ? 
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How well program serves following purposes 



Provide information 
to indicate 
educational status 
in Nevada 



NOW SERVE 



Not At 
All 
Well 



Very 
Well 



SHOULD SERVE 



Not At 
All 
Well 



Very 
Well 



Other (specify below) 



1 2 3 4 5 ? 



1 2 3 4 5 



3. Information obtained through the Nevada Proficiency Examination Program can be useful to various 
groups of people. Some of these groups are listed in the left column. In your opinion, how useful is the 
information to each of these groups ngw? Ideally, how useful should the information be to these groups? 
Circle a number under the respective columns. Circle "?" if you are not sure or are unable to provide a 
response. 



Usefulness of program information to following groups 



NOW 



Not At 

Ail 
Useful 



Very 
Useful 



SHOULD BE 



Not At 

All 
Useful 



Very 
Useful 



Students 

Parents 

Local school 
boards 

Teachers 

Principals 

District 

superintendents 

Curriculum 
personnel 

State 

Superintendent 
of Instruction 

State Board of 
Education 

Legislature 

Media 

Public 

Other (specify below) 



2 3 4 5 ? 

2 3 4 5 ? 

2 3 4 5 ? 

2 3 4 5 ? 

2 3 4 5 ? 

^ 3 4 5 ? 

2 3 4 5 ? 

2 3 4 5 ? 



2 3 

2 3 

2 3 

2 3 



4 5 ? 

4 5 ? 

4 5 ? 

4 5 ? 



2 3 4 

2 3 4 



2 
2 
2 
2 



3 
3 
3 
3 



4 
4 
4 
4 



5 ? 
5 ? 



2 3 4 5 ? 

2 3 4 5 ? 

2 3 4 5 ? 

2 3 4 5 ? 

2 3 4 5 ? 



5 ? 

5 ? 

5 ? 

5 ? 
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4. In your opinion, what kinds of test scores should be included in reports provided by the Nevada 
Proficiency Examination Program? Circle "?" if you are not sure. 



Definitely 
No 



Pass/fail rates 

Raw Scores 

Percents correct 

Percentiles 

Scale scores 

Normal Curve 
Equivalents 

Stanines 

Grade equivalents 
State averages 
District averages 
Building level averages 
Classroom averages 
Other (specify below) 



2 
2 
2 
2 
2 

2 
2 
2 
2 
2 
2 
2 



3 
3 
3 
3 
3 

3 
3 
3 
3 
3 
3 
3 



4 
4 
4 
4 
4 

4 
4 
4 
4 
4 
4 
4 



Definitely 
Yes 

5 
5 
5 
5 
5 



5 
5 
5 
5 
5 



5. Currently the Nevada Proficiency Examination Program assesses student performance at grades 3, 6, 9 and 
11 in reading, mathematics and writing. In your opinion, which grade levels and subject areas should be 
included in the Program? Circle "?" if you are not sure. 



Grade Level: 

Kindergarten 

First 

Second 

Third 

Fourth 

Fifth 

Sixth 

Seventh 

Eighth 

Ninth 

Tenth 

Eleventh 

Twelfth 
Subject Area: 

Reading 

Mathematics 

Language Arts 

Writing 

Science 

Social Studies 

Thinking Skills 

Other (specify below) 



Definitely 
No 



Definitely 
Yes 



2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 
2 

2 
2 
2 
2 
2 
2 
2 



3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 
3 

3 
3 
3 
3 
3 
3 
3 



4 
4 
4 
4 
4 
4 



4 
4 
4 
4 
4 
4 
4 



5 
5 
5 
5 
5 
5 
5 
5 
5 
5 
5 
5 
5 

5 
5 
5 
5 
5 
5 
3 



? 
? 
? 
? 
? 

? 
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6. How satisfied are you with respect tc the following aspects of the Nevada Proficiency Examination 
Program? Circle T if you are not sure or are unable to provide a response. 



Overall purpose(s) 

Specific competencies 
included in program 

Graduation standards 

Measurement instruments 

Grade levels tested 

Subject areas tested 

Scoring of writing samples 

Frequency of testing 

Frequency of reporting 

Clarity and usefulness 
of test results 

Ways of handling 

(a) Private school 

students 

(b) Home-schooled 

students 

(c) Language different 

students 



(d) Special education 
students 

Program funding level 

Fiscal arrangements 

The joint administration 
of the program by the 
State Department of 
Education and local 
districts 



Not At All 
Satisfied 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 7 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 

1 2 3 



Very 
Satisfied 

4 5 ? 

4 5 ? 

4 5 ? 

4 5 ? 

4 5 ? 

4 5 ? 

4 5 ? 

4 5 ? 

4 5 ? 

4 5 ? 

4 5 ? 

4 5 ? 

4 5 ? 

4 5 ? 

4 5 ? 

4 5 ? 

4 5 ? 



Not At All 
Satisfied 



Very 
Satisfied 



Dissemination of program 
information to the 
public 

Use of standardized 
achievement tests 

Use of state developed 
proficiency tests 

Ways in which data 
provided by the program 
are currently used 

Congruency between 
objectives/competencies 
specified in the program 
and local school curriculum: 

(a) Reading 

(b) Mathematics 

(c) Writing 

Program impact on the 
instructional process: 

(a) Reading 

(b) Mathematics 

(c) Writing 



1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 



1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 



7. Information obtained through the Nevada Proficiency Examination Program can be used in many different 
ways. Some of the possible uses are listed in the left-hand column. Based on your experience, to what 
extent is the information used in any of these ways now? Ideally, to what extent should the information be 
used in these wa>s? Circle "?" if you are not sure or are unable to provide a response. 



Extent of use of program information in following ways 

NOW SHOULD BE 



Public 

accountability 

Curriculum 
improvement 
at the state level 

Monitoring 

achievement 

trends 

Influencing 
educational 
policy 

Comparisons with 
national norms 

Comparisons among 
districts 

Comparisons among 

schools 

Incentives 

Sanctions 

Curriculum 
improvement at 
the local level 

Increased 
attention to 
teaching basic 
skills 

Other (specify below) 



Not To a 

At Great 
All Extent 



1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 



Not 
At 
All 



To a 
Great 
Extent 



1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 

1 2 3 4 5 ? 



12 3 4 



1 2 3 4 5 ? 



1 2 3 4 5 



1 2 3 4 5 ? 



8. Information obtained through the Nevada Proficiency Examination Program can be used to make various 
comparisons with regard to student achievement. Would you be in favor of using the information to make 
the following comparisons? Circle T if you are not sure or are unable to provide a response. 



Definitely 
No 



Definitely 
Yes 



Comparing Nevada average 
with national norms 

Comparing one county 
wi th another without 
considering demographics 

Comparing one school 
with another without 
considering demographics 

Comparing one classroom 
with another 

Comparing demographically 
similar schools 

Comparing demographically 
similar counties 

Other comparison (specify below) 
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9. In your judgment, would the following steps improve the Nevada Proficiency Examination Program 
Circle T if you are not sure or are unable to provide a response. 



Definitely Definitely 
No Yes 



Establish a link with 
state school improvement 
programs 

Establish a link with 
the accreditation of 
public schools 

Randomly monitor proficiency 
testing sites to ensure 
proper test administration 

Have all proficiency tests 
scored by an outside 
contractor 

Have all proficiency tests 
scored by testing personnel 
at the State Department of 
Education 

Conduct a statewide annual 
testing conference to share 
results 

Report proficiency test 
results by subgroup 
(e.g., by gender and 
minority group) 

Use a standardized 
achievement test 
for all grade 
levels 

Use state develope 1 tests 
for all grade levels 

Increase public information 
materials provided by the 
program 



10. 



Please let us know how you feel about the state developed Nevada high school proficiency tests by 
indicating how strongly you agree or disagree with each of the following statements. Circle the number 
that best indicates how yoj feel. Circle "?" if you are not sure or are unable to provide a response. 



9 

ERIC 



Strongly Strongly 
Disagree Agree 

(a) The test content is 

appropriate. 1 2 3 4 5 

(b) The tests are too easy. 1 2 3 4 5 

(c) The tests are too difficult. 1 2 3 4 5 

(d) The tests are just right in 

terms of difficulty level. 1 2 3 4 5 

(e) The format of the tests 
(e.g., print size, item 
placement, paper stock) 

is satisfactory. 1 2 3 4 5 

(f) The tests separate 
functional from non- 
functional students 
(i.e., only functional 
students will pass the 

test). 1 2 3 4 5 
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11. Based on your experience with the Nevada Proficiency Examination Program, what are some of the 

unanticipated program effects? Please include both positive and negative effects. Use i everse side if you 
need more space. 



12. In the space below, please provide any comments or suggestions that would help the Nevada Department 
of Education improve the Nevada Proficiency Examination Program. Use reverse side if you need more 
space. 



Please return the completed questionnaire in the enclosed self-addressed, stamped envelope to: 

Dr. Kevin Crowe, Director 
Planning, Research and Evaluation 
Nevada Department of Education 
400 W. King Street 
Carson City, NV 89710 



THANK YOU VERY MUCH! 



Nevada Proficiency Examination Program 
Simple Interview Questions 

For selected members of the following respondent groups: 

Local school boards 
State Board of Education 
Legislature 

State Department of Education 

1. In your view, what have been the primary goals of tie Nevada Proficiency Examination Program? Do 
you think these goals should be changed? 

2. Who should set policies for the program? 

3. What roles should the local districts play in the policy-making process? 

4. At present, the program is paid for by the local districts. Should the program be funded that way? If no, 
how should it be funded? 

5. In your opinion, who (which agency) should administer the program? 

6. What do you think of the state developed proficiency tests in reading, mathematics and writing? For 
example, do you think they are too easy, too difficult, or just about right? 

7. How should the proficiency test data be aggregated and reported (e.g., by county, by school)? 

8. Who should have access to the proficiency test data? 

9. Is there any evidence that th; program has influenced the curriculum and instruction at the school level? 
If yes, in what ways? 

10. In what ways has the program enhanced (or hindered) student achievement in Nevada? 

11. Should the program be expanded to cover more grades and/or more subject areas? Why do you think so? 

12. To what exten e the specific competencies specified in the program congruent with school curriculum 
and instruction: 

13. In your opinion, does the program provide information that is useful to the various groups it :s intended to 
serve (e.g., State Board of Education, legislature, Nevada Department of Education, county /school level 
personnel)? 

14. What suggestions do you have for improving the program? 



9 

ERIC 



9 

ERIC 



Nevada Proficiency Examination Program 
Sample Interview Questions 

For: 

Business leaders 
Community members 

1. What do you know about the Nevada Proficiency Examination Program? 

2. Do you think it is a good idea to have the program? Why do you think so? 

3. In what ways has the program helped the children? 

4. What do you think of the state developed proficiency tests in reading, mathematics and writing? For 
example, do you think they are too easy, too difficult, or just about right? 

5. What are some of the things you like to see changed in the program? 

6. What suggestions do you have for improving the program? 

7. Do you have any other comments about the program? 



Nevada Proficiency Examination Program 
Sample Questionnaire Items 

(To be incorporated in state developed proficiency tests) 

Please let us know how you feel about the state developed high school proficiency tests. Tell us how you feel by 
indicating how strongly you agree or disagree with each of the following statements. Mark your responses in 
Block 8 - SPECIAL, items 1 through 9, on your answer sheet. 



1. The reading test is too 
easy. 

2. The format (e.g., print 
size, item placement, 
paper stock) of the 
reading test is 
satisfactory. 

3. The reading test measures 
skills and knowledge 

that are taught in our 
school. 

4. The reading test helps 
provide a good basis for 
determining who should 
receive a high school 
diploma. 



Strongly 
Disagree 



Strongly 
Agree 



5. The mathematics test 
is too easy. 

6. The format (e.g., print 
size, item placement, 
paper stock) of the 
mathematics test is 
satisfactory. 

7. The mathematics test 
measures skills and 
knowledge that are 
taught in our school. 

8. The mathematics test 
helps provide a good 
basis for determining 
who should receive a 
high school diploma. 



a 



The writing test is too 
easy. 



Nevada Proficiency Examination Program 
Sample Questionnaire Items 

(To be used in the Department's bi-annual telephone poll of parents (Needs Assessment: The Opinions cf 
Nevada Voters About Their Public Schools) in spring, 1990) 

Directions: For each item, please circle the number that best reflects your opinion cr perception. 

1. How much do you know about the Nevada Proficiency Examination Program? 

Very little 1 2 3 4 5 A lot 

2. In your judgment, has the program had any effects on the quality of education in Nevada? 

Definitely no 1 2 3 4 5 Definitely yes 

3. Would you agree that the state developed proficiency tests do a good job in separating functional from 
non-functional high school students? 

Strongly Strongly 
Disagree 1 2 3 4 5 Agree 

4 In general, how would you rate the Nevada Proficiency Examination Program? 

Very poor 1 2 3 4 5 Excellent 
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